Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: split stages in categories #25112

Merged
merged 9 commits into from
Apr 23, 2021
Merged

CI: split stages in categories #25112

merged 9 commits into from
Apr 23, 2021

Conversation

v1v
Copy link
Member

@v1v v1v commented Apr 15, 2021 •
edited
Loading

What does this PR do?

Split the pipeline in four meta-stages:

  1. the linting -> requires 21 workers for each merge commit (branches)
  2. the build/test for the core platforms (linux, windows-2019 and arm) -> requires 41 workers for each merge commit (branches)
  3. the build/test for the extended platforms (windows <=2016, macOS) -> requires 113 workers for each merge commit (branches)
  4. the packaging -> requires 30 workers for each merge commit (branches)

Why is it important?

Potentially, this will reduce the CI load by provisioning less number of workers in parallel, which at the moment is causing issues with timeouts. This should help to smooth the required CI workers in batches and affecting a bit less the other pipelines.

Issues

Superseedes #24790

UI

If failures in one of the stages in the build/test meta-stage

image

If no failures but caching the linting:

image

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021
@v1v v1v self-assigned this Apr 15, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 15, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: v1v commented: /test

  • Start Time: 2021-04-21T18:01:00.012+0000

  • Duration: 95 min 10 sec

  • Commit: ea302e2

Test stats 🧪

Test Results
Failed 0
Passed 32331
Skipped 4323
Total 36654

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 32331
Skipped 4323
Total 36654

@v1v v1v mentioned this pull request Apr 15, 2021
Merged
v1v
v1v commented Apr 15, 2021
View reviewed changes
Jenkinsfile Outdated Show resolved Hide resolved
@v1v
Copy link
Member Author

v1v commented Apr 15, 2021

/test

@v1v
Copy link
Member Author

v1v commented Apr 19, 2021

/test

4 similar comments
@v1v
Copy link
Member Author

v1v commented Apr 19, 2021

/test

@v1v
Copy link
Member Author

v1v commented Apr 19, 2021

/test

@v1v
Copy link
Member Author

v1v commented Apr 20, 2021

/test

@v1v
Copy link
Member Author

v1v commented Apr 20, 2021

/test

@v1v v1v marked this pull request as ready for review April 20, 2021 15:03
@v1v v1v requested review from a team as code owners April 20, 2021 15:03
@v1v v1v added automation backport-v7.12.0 Automated backport with mergify backport-v7.13.0 Automated backport with mergify Team:Automation Label for the Observability productivity team labels Apr 20, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 20, 2021
@urso
Copy link

urso commented Apr 20, 2021

Why does the lint phase take that many workers?

@v1v
Copy link
Member Author

v1v commented Apr 20, 2021

Why does the lint phase take that many workers?

The linting was split to run specifically per beat, therefore there are 21 lint stages in total defined in each Jenkinsfile.yml. See https://github.com/elastic/beats/search?l=YAML&q=%22Lint%22

This is not something new but something that was added last year -> #22103

We could move away from this approach and use the previous behaviour, aka make check, in favour of using the 1 worker instead of 21.

What do you think?

@v1v v1v changed the title PoC: split mandatory from other stages (2) CI: split stages in categories Apr 20, 2021
@kuisathaverat
Copy link
Contributor

the old checkout takes 15 min always, launch the lint in parallel will launch 21 workers only if you modify libbeat or something build everything is the worst case, fo one beat will launch 1-2 and take 5 min or less, I think that worth the quick feedback on linting.

cachedout
cachedout approved these changes Apr 21, 2021
View reviewed changes
Jenkinsfile Outdated Show resolved Hide resolved
@v1v
Copy link
Member Author

v1v commented Apr 21, 2021

/test

@v1v v1v added the backport-v7.14.0 Automated backport with mergify label Apr 22, 2021
@v1v v1v requested review from urso and jsoriano April 22, 2021 08:45
@urso
Copy link

urso commented Apr 22, 2021

the old checkout takes 15 min always, launch the lint in parallel will launch 21 workers only if you modify libbeat or something build everything is the worst case, fo one beat will launch 1-2 and take 5 min or less, I think that worth the quick feedback on linting.

Let's keep it for now then. We should keep this one in mind, in case we need to 'optimize' more.

Hm, this makes me wonder if:

  1. we can make the lint step more selective. Do we need to lint Filebeat, even if the change is in libbeat? I would consider that all lints are 'local' to the actual product
  2. we can improve make check itself to only run the checks on a preconfigured set of sub-projects (based on environment variable the selective testing support defines).

@v1v v1v merged commit f4c570b into elastic:master Apr 23, 2021
@v1v v1v mentioned this pull request Apr 23, 2021
Closed
@v1v v1v deleted the feature/split-stages-2 branch April 23, 2021 08:10
@v1v
Copy link
Member Author

v1v commented Apr 23, 2021

I raised #25257 to keep track of the previous conversation

mergify bot pushed a commit that referenced this pull request Apr 23, 2021
@v1v
CI: split stages in categories (#25112)
917d2dd 
(cherry picked from commit f4c570b)
@mergify mergify bot mentioned this pull request Apr 23, 2021
Merged
mergify bot pushed a commit that referenced this pull request Apr 23, 2021
@v1v
CI: split stages in categories (#25112)
f186e23 
(cherry picked from commit f4c570b)
@mergify mergify bot mentioned this pull request Apr 23, 2021
Merged
mergify bot pushed a commit that referenced this pull request Apr 23, 2021
@v1v
CI: split stages in categories (#25112)
bcdbbd6 
(cherry picked from commit f4c570b)

# Conflicts:
#	x-pack/metricbeat/Jenkinsfile.yml
#	x-pack/osquerybeat/Jenkinsfile.yml
@mergify mergify bot mentioned this pull request Apr 23, 2021
Closed
v1v added a commit that referenced this pull request Apr 30, 2021
@mergify @v1v
CI: split stages in categories (#25112) (#25258)
1114678 
(cherry picked from commit f4c570b)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
v1v pushed a commit that referenced this pull request Jun 29, 2021
@mergify
CI: split stages in categories (#25112) (#25259)
10bb907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kuisathaverat kuisathaverat kuisathaverat approved these changes

@cachedout cachedout cachedout approved these changes

@urso urso urso approved these changes

@jsoriano jsoriano Awaiting requested review from jsoriano

Assignees

@v1v v1v

Labels
automation backport-v7.12.0 Automated backport with mergify backport-v7.13.0 Automated backport with mergify backport-v7.14.0 Automated backport with mergify Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@v1v @elasticmachine @urso @kuisathaverat @cachedout